Demonstrational Techniques for Instructible User Interface Agents

But what is this "environment"? For the majority of applications used by most mainstream computer users, such as spreadsheets, text editors, mail readers, graphical editors, CAD programs, etc., a direct-manipulation graphical interface constitutes the "environment" with which a software agent must interact in order to be ultimately useful to the user. The program might have internal representations of objects of interest to the user [as frames, propositions, neural nets, or specialized data structures], and perform inference on these representations, but the sensors must detect actions taken by the user in the interface, and the effectors must deliver effects that are ultimately visible or audible in the user interface.